Information and exponential families

Chernoff information of exponential families

Chernoff information upper bounds the probability of error of the optimal Bayesian decision rule for 2-class classification problems. However, it turns out that in practice the Chernoff bound is hard to calculate or even approximate. In statistics, many usual distributions, such as Gaussians, Poissons or frequency histograms called multinomials, can be handled in the unified framework of expone...

Distributed Estimation, Information Loss and Exponential Families

Distributed learning of probabilistic models from multiple data repositories with minimum communication is increasingly important. We study a simple communication-efficient learning framework that first calculates the local maximum likelihood estimates (MLE) based on the data subsets, and then combines the local MLEs to achieve the best possible approximation to the global MLE given the whole d...

Exponential families

Graphical Models and Exponential Families

We provide a classi cation of graphical models according to their representation as subfamilies of exponential families. Undirected graphical models with no hidden variables are linear exponential families (LEFs), directed acyclic graphical models and chain graphs with no hidden variables, including Bayesian networks with several families of local distributions, are curved exponential families ...

Optimally approximating exponential families

This article studies exponential families E on finite sets such that the information divergence D(P E) of an arbitrary probability distribution from E is bounded by some constant D > 0. A particular class of low-dimensional exponential families that have low values of D can be obtained from partitions of the state space. The main results concern optimality properties of these partition exponent...